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Abstract 

Model  checking  is  an  automatic  technique  for  verifying  sequential  circuit  designs  and  pro¬ 
tocols.  An  efficient  search  procedure  is  used  to  determine  whether  or  not  the  specification 
is  satisfied.  If  it  is  not  satisfied,  our  technique  will  produce  a  counterexample  execution 
trace  that  shows  the  cause  of  the  problem.  Although  finding  counterexamples  is  extremely 
important,  there  is  no  description  of  how  to  do  this  in  the  literature  on  model  checking.  We 
describe  an  efficient  algorithm  to  produce  counterexamples  and  witnesses  for  symbolic  model 
checking  algorithms.  This  algorithm  is  used  in  the  SMV  model  checker  and  works  quite  well 
in  practice.  We  also  discuss  how  to  extend  our  technique  to  more  complicated  specifications. 
This  extension  makes  it  possible  to  find  counterexamples  for  verification  procedures  based 
on  showing  language  containment  between  various  types  of  u;-automata. 
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1.  Introduction 


Complex  state-transition  systems  occur  frequently  in  the  design  of  sequential  circuits  and 
protocols.  During  the  past  ten  years,  researchers  at  Carnegie  Mellon  University  have  devel¬ 
oped  an  alternative  approach  to  verification  called  temporal  logic  model  checking  [5,  6].  In 
this  approach  specifications  are  expressed  in  a  propositional  temporal  logic,  and  circuit  de¬ 
signs  and  protocols  are  modeled  as  state-transition  systems.  An  efficient  search  procedure  is 
used  to  determine  automatically  if  the  specifications  are  satisfied  by  the  transition  systems. 

One  of  the  most  important  advantages  of  model  checking  over  mechanical  theorem  provers 
or  proof  checkers  for  verification  of  circuits  and  protocols  is  its  counterexample  facility. 
Typically,  the  user  provides  a  high  level  representation  of  the  model  and  the  specification  to 
be  checked.  The  model  checking  algorithm  either  terminates  with  the  answer  true,  indicating 
that  the  model  satisfies  the  specification,  or  gives  a  counterexample  execution  that  shows 
why  the  formula  is  not  satisfied.  The  counterexamples  can  be  essential  in  finding  subtle 
errors  in  complex  designs. 

The  main  disadvantage  of  model  checking  is  the  state  explosion  which  can  occur  if  the  sys¬ 
tem  being  verified  has  many  components  that  can  make  transitions  in  parallel.  Recently,  the 
size  of  the  transition  systems  that  can  be  verified  by  model  checking  techniques  has  increased 
dramatically  after  the  introduction  of  ordered  binary  decision  diagrams  (OBDDs)  [2].  By 
applying  this  technique,  verification  of  systems  that  have  more  than  lO100  states  has  become 
possible  [3,  11].  However,  finding  counterexamples  is  significantly  more  difficult  when  OB¬ 
DDs  are  used  in  model  checking  instead  of  explicit  state  enumeration  techniques,  especially 
when  fairness  constraints  are  involved. 

Although  finding  counterexamples  is  extremely  important,  as  far  as  we  know,  there  is  no 
description  of  how  to  do  this  in  the  literature  on  model  checking.  In  this  paper,  we  describe  an 
efficient  algorithm  to  produce  counterexamples  and  witnesses  for  model  checking  algorithms. 
The  algorithm  is,  in  fact,  the  one  that  is  used  in  the  SMV  model  checker  developed  at 
Carnegie  Mellon  [11]  and  works  quite  well  in  practice.  We  show  how  the  counterexample 
facility  can  be  used  to  debug  a  subtle  asynchronous  circuit  design.  We  also  discuss  how 
to  extend  our  technique  to  more  complicated  temporal  formulas.  This  extension  makes 
it  possible  to  find  counterexamples  for  verification  procedures  based  on  showing  language 
containment  between  various  types  of  (u-automata. 

This  paper  is  organized  as  follows:  The  properties  of  OBDDs  that  we  need  are  briefly 
discussed  in  Section  2.  The  next  section  describes  the  temporal  logic  CTL  that  we  use  tor 
specifying  properties  of  sequential  circuits  and  protocols.  Section  4  explains  the  symbolic 
model  checking  algorithm  for  CTL,  and  Section  5  shows  how  fairness  constraints  can  be 
handled.  Section  6  is  the  main  section  of  the  paper.  We  describe  how  counterexamples 
and  witnesses  are  generated.  We  also  give  an  example  that  shows  how  this  facility  can  be 
used  in  sequential  circuit  verification.  In  the  next  section  we  extend  the  counterexample 
facility  to  a  wider  class  of  temporal  properties.  Section  8  describes  how  our  techniques  can 
be  used  to  generate  counterexamples  for  verification  procedures  that  are  based  on  showing 
inclusion  between  cu-automata.  The  paper  concludes  in  Section  9  with  a  discussion  of  possible 
directions  for  future  research. 
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2.  Binary  Decision  Diagrams 


Ordered  binary  decision  diagrams  (OBDDs)  are  a  canonical  form  representation  for  boolean 
formulas  [2].  They  are  often  substantially  more  compact  than  traditional  normal  forms  such 
as  conjunctive  normal  form  and  disjunctive  normal  form,  and  they  can  be  manipulated  very 
efficiently.  An  OBDD  is  similar  to  a  binary  decision  tree,  but  has  the  following  properties: 

•  Its  structure  is  a  directed  acyclic  graph  rather  than  a  tree. 

•  Variables  occur  in  the  same  order  on  every  path  from  root  to  leaf. 

•  No  two  subgraphs  in  the  graph  represent  the  same  function. 

For  a  given  variable  ordering,  the  OBDD  representation  of  a  boolean  formula  is  unique  [2]. 

We  can  perform  most  logical  operations  efficiently  using  OBDDs.  The  function  that 
restricts  some  argument  xt  of  a  boolean  function  /  to  a  constant  value  6,  denoted  by  /  | 
can  be  performed  in  time  linear  in  the  size  of  the  original  OBDD  [2].  The  restriction 
algorithm  allows  us  to  compute  the  OBDD  for  the  formula  3xf  as  /  |j,_o  V/  |x^i.  All 
16  two-argument  logical  operations  can  be  implemented  efficiently  on  boolean  functions 
that  are  represented  as  OBDDs.  The  complexity  of  these  operations  is  linear  in  the  size  of 
the  argument  OBDDs  [2].  Moreover,  equivalence  of  two  boolean  functions  can  be  decided  in 
constant  time  [1]. 

OBDDs  are  used  in  this  paper  for  obtaining  concise  representations  of  relations  over  finite 
domains  [3].  If  R  is  n-ary  relation  over  {0, 1}  then  R  can  be  represented  by  the  OBDD  of 
its  characteristic  function 


Jr{x  i,...,xn)  =  1  iff  R(xi, . . .  ,xn). 

If  R  is  an  n-ary  relation  over  the  finite  domain  D  with  |D|  >  2,  R  can  still  be  represented 
as  an  OBDD  if  an  appropriate  binary  encoding  is  used  for  D. 


3.  The  temporal  logic  CTL 

The  logic  that  we  use  to  specify  circuits  is  a  propositional  temporal  logic  of  branching  time, 
called  CTL  or  Computation  Tree  Logic  [6].  In  this  logic  each  of  the  usual  forward-time 
operators  of  linear  temporal  logic  (G  globally  or  invariant ly ,  F  sometime  in  tin  future,  X 
nexttime  and  U  until)  must  be  directly  preceded  by  a  path  quantifier.  The  path  quantifier 
can  either  be  an  A  (for  all  computation  paths)  or  an  E  (for  some  computation  path).  Thus, 
some  typical  CTL  formulas  are  AG  /,  which  holds  in  a  state  provided  that  /  holds  globally 
along  all  possible  computation  paths  starting  from  that  state,  and  EF  /,  which  holds  in  a 
state  provided  that  there  is  a  computation  path  such  that  /  holds  in  the  future  on  the  path. 

In  order  to  explain  our  verification  procedure,  it  is  convenient  to  express  the  CTL  opera¬ 
tors  with  universal  path  quantifiers  in  terms  of  the  operators  with  existential  path  quantifiers, 
taking  advantage  of  the  duality  between  universal  and  existential  quantification.  Conse¬ 
quently,  in  our  description  of  the  syntax  and  semantics  of  CTL,  we  specify  the  existential 


path  quantifiers  directly  and  treat  the  universal  path  quantifiers  as  syntactic  abbreviations. 
Let  P  be  the  set  of  atomic  propositions,  then: 

1.  Every  atomic  proposition  p  in  P  is  a  formula  in  CTL. 

2.  If  /  and  g  are  CTL  formulas,  then  so  are  ->/,  /  V  g,  EX/,  E[/  U  g]  and  EG  /. 

The  semantics  of  a  CTL  formula  is  defined  with  respect  to  a  labeled  state-transition 
graph.  A  labeled  state-transition  graph  is  a  5-tuple  M  =  (AP,  S,L,  N,  Sg)  where  AP  is  a 
set  of  atomic  propositions,  S  is  a  finite  set  of  states,  L  is  a  function  labeling  each  state  with 
a  set  of  atomic  propositions,  N  C  S  x  S  is  a  transition  relation,  and  5o  is  a  set  of  initial 
states.  A  computation  path  is  an  infinite  sequence  of  states  So,  si,  S2,  ■  •  •  such  that  N(si,  Si+i ) 
is  true  for  every  i. 

The  propositional  connectives  ->  and  V  have  their  usual  meanings  of  negation  and  disjunc¬ 
tion.  The  other  propositional  operators  can  be  defined  in  terms  of  these.  X  is  the  nexttime 
operator:  EX  /  will  be  true  in  a  state  s  of  M  if  and  only  if  s  has  a  successor  s'  such  that 
/  is  true  at  s'.  U  is  the  until  operator:  E[/  U  g\  will  be  true  in  a  state  s  of  M  if  and  only 
if  there  exists  a  computation  path  starting  in  s  and  an  initial  prefix  of  the  path  such  that 
g  holds  at  the  last  state  of  the  prefix  and  /  holds  at  all  other  states  along  the  prefix.  The 
operator  G  is  used  to  express  the  invariance  of  some  property  over  time:  EG  /  will  be  true 
at  a  state  s  if  there  is  a  path  starting  at  s  such  that  /  holds  at  each  state  on  the  path.  If /is 
true  in  state  s  of  structure  M,  we  write  M,  s  J=  /.  A  CTL  formula  j  is  identified  with  the 
set  {s|M,s  |=  /}  of  states  that  make  /  true.  We  use  the  following  syntactic  abbreviations 
for  CTL  formulas: 

•  AX  /  =  EX  ->/  which  means  that  /  holds  at  all  successor  states  of  the  current  state 
(/  must  hold  at  the  next  state). 

•  EF  /  =  E[true  U  f]  which  means  that  for  some  path,  there  exists  a  state  on  the  path 
at  which  /  holds  (/  is  possible  in  the  future). 

•  AF  /  ==  — 1  EG  -if  which  means  that  for  every  path,  there  exists  a  state  on  the  path  at 
which  /  holds  (/  is  inevitable  in  the  future). 

•  AG  /  =  -1  EF  —>f  which  means  that  for  every  path,  J  holds  in  each  state  on  the  path 
(/  holds  globally  along  all  paths). 

•  A[/  U  g]  =  ->  E[-\<7  U  -1/  A  ->g\  A  ^  EG  -></  which  means  that  for  every  path,  there 
exists  an  initial  prefix  of  the  path  such  that  g  holds  at  the  last  state  ot  the  pielix  and 
/  holds  at  all  other  states  along  the  prefix  (/  holds  until  g  holds,  along  all  paths). 

4.  Symbolic  Model  Checking 

Model  checking  is  the  problem  of  finding  the  set  of  states  in  a  state-transition  graph  where  a 
given  CTL  formula  is  true.  There  is  a  program  called  EMC  (Extended  Model  Checker)  that 
solves  this  problem  using  efficient  graph-traversal  techniques.  If  the  model  is  represented  as 
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a  state-transition  graph,  the  complexity  of  the  algorithm  is  linear  in  the  size  of  the  graph 
and  in  the  length  of  the  formula.  The  algorithm  is  quite  fast  in  practice  [5,  6].  However,  an 
explosion  in  the  size  of  the  model  may  occur  when  the  state-transition  graph  is  extracted 
from  a  finite  state  concurrent  system  that  has  many  processes  or  components. 

In  this  section,  we  describe  a  symbolic  model  checking  algorithm  for  C'TL  which  uses 
OBDDs  to  represent  the  state-transition  graph.  Assume  that  the  behavior  of  the  concurrent 
system  is  determined  by  n  boolean  state  variables  i’i,v2, . . . ,  vn.  The  transition  relation 
R(v,  v')  for  the  concurrent  system  is  given  as  a  boolean  formula  in  terms  of  two  copies  of 
the  state  variables:  v  =  (iq, . . . ,  vn)  which  represents  the  current  state  and  v'  =  (t/, . . .  ,v'n) 
which  represents  the  next  state.  The  formula  R(B,v')  is  now  converted  to  an  OBDD.  This 
usually  results  in  a  very  concise  representation  of  the  transition  relation. 

Our  model  checking  algorithm  is  based  on  the  standard  fixpoint  characterizations  of  the 
basic  CTL  operators.  A  fixpoint  of  r  :  2s  — >  2s  is  a  set  S'  C  S  such  that  t(S')  =  S'.  If  r 
is  monotonic,  it  has  a  fixpoint  So  that  is  a  subset  of  every  other  fixpoint  of  r.  So  is  called 
the  least  fixpoint  of  r  and  is  denoted  by  lfp  f[r(f)].  The  greatest  fixpoint  of  r,  gfp/[r(/)], 
can  be  defined  similarly  as  the  fixpoint  of  t  that  is  a  superset  of  all  other  fixpoints.  It,  can 
be  shown  that  the  least  fixpoint  lfp  f[r(f)]  is  the  limit  of  the  sequence  of  approximations 

False,  r(False),  r2(False), . . . ,  r!(False), . . . 

and  the  greatest  fixpoint  gfp  /[r(/)]  is  the  limit  of  the  sequence  of  approximations 

True,  r(True),  r2(True), . . . ,  r‘(True), . . . 

When  the  state-transition  graph  is  finite,  both  of  these  sequences  are  guaranteed  to  converge 
in  a  finite  number  of  steps. 

Each  of  the  basic  CTL  operators  can  be  characterized  as  a  least  or  greatest  fixpoint  of 
some  functional  r  :  2s  — >  25.  In  particular,  it  is  shown  in  [5]  that 

•  E[/  U  g}  =  lfp  Z  \g  V  (/  A  EX  Z)],  and 

•  EG  /  =  gfp  Z  [/  A  EX  Z], 

The  symbolic  model  checking  algorithm  is  implemented  by  a  procedure  Check  t  hat  takes 
the  CTL  formula  to  be  checked  as  its  argument  and  returns  an  OBDD  that  represents  exactly 
those  states  of  the  system  that  satisfy  the  formula.  Of  course,  the  output  of  Check  depends 
on  the  system  being  checked;  this  parameter  is  implicit  in  the  discussion  below.  We  define 
Check  inductively  over  the  structure  of  CTL  formulas.  If  /  is  an  atomic  proposition  e,,  then 
Check(f )  is  simply  the  OBDD  for  vt.  Formulas  of  the  form  EX/,  E[/U</],  and  EG  /  are 
handled  by  the  procedures: 

Check  ("EX.  /)  =  Check  EX  {Check  {/ )), 

Check{E[fU  g])  =  CheckEU {Check{f),  Chec.k{g)), 

Check  {EG  f)  =  CheckEC(Check(f)). 

Notice  that  these  intermediate  procedures  take  boolean  formulas  as  their  arguments,  while 
Check  takes  a  CTL  formula  as  its  argument.  CTL  formulas  of  the  form  /  V  g  or  -> f  are 
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handled  using  the  standard  algorithms  for  computing  boolean  connectives  with  OBDDs. 
Since  AX/,  A[/  U  g]  and  AG  /  can  all  be  rewritten  using  just  the  above  operators,  this 
definition  of  Check  covers  all  CTL  formulas. 

The  procedure  for  CheckEX  is  straightforward  since  the  formula  EX  /  is  true  in  a  state 
if  the  state  has  a  successor  in  which  /  is  true. 

CheckEX (f(v))  =  3v'  [f(v')  A  R(v,v')}. 

If  we  have  OBDDs  for  /  and  R ,  then  we  can  compute  an  OBDD  for 

3v'[f(v')AR(v,v')}. 

using  the  BDD  operations  given  in  Section  2. 

The  procedure  for  CheckEU  is  based  on  the  least  fixpoint  characterization  for  the  CTL 
operator  EU. 

CheckEU (f(v),g(v))  =  lfp  Z(v)  [g(v)  V  (/(h)  A  CheckEX  {Z(v)))\. 

In  this  case  we  can  compute  the  sequence  of  approximations 

Qoi  Qii  •  •  •  iQii  •  •  • 

for  the  least  fixpoint  as  described  above.  If  we  have  OBDDs  lor  /,  g ,  and  the  current 
approximation  Qi,  then  we  can  compute  an  OBDD  for  the  next  approximation  Qi+l.  Since 
OBDDs  provide  a  canonical  form  of  boolean  functions,  it  is  easy  to  test  for  convergence  by 
comparing  consecutive  approximations.  When  Qi  =  Qi+u  this  process  terminates.  The  set 
of  states  corresponding  to  E[/  U  g]  will  be  represented  by  the  OBDD  lor  Qi. 

CheckEG  is  similar.  In  this  case  the  procedure  is  based  on  the  greatest  fixpont  charac¬ 
terization  for  the  CTL  operator  EG 

CheckEG(f(v))  =  gfp  Z(v)  [/(h)  A  CheckEX  (Z(v))\. 

If  the  OBDD  for  /  is  given,  then  the  sequence  of  approximations  for  the  greatest  fixpoint 
can  be  used  to  compute  the  OBDD  representation  for  the  set  of  states  that  satisfy  EG  f . 


5.  Fairness  Constraints 

Next,  we  consider  the  issue  of  fairness.  In  many  cases,  we  are  only  interested  in  the  correct¬ 
ness  along  fair  computation  paths.  For  example,  if  we  are  verifying  an  asynchronous  circuit 
with  an  arbiter,  we  may  wish  to  consider  only  those  executions  in  which  the  arbiter  does  not 
ignore  one  of  its  request  inputs  forever.  This  type  of  property  cannot  be  expressed  directly 
in  CTL.  In  order  to  handle  such  properties  we  must  modify  the  semantics  of  CTL  slightly. 
A  fairness  constraint  can  be  an  arbitrary  set  of  states,  usually  described  by  a  formula  of 
the  logic.  A  path  is  said  to  be  fair  with  respect  to  a  set  ol  fairness  constraints  if  each 
constraint  holds  infinitely  often  along  the  path.  The  path  quantifiers  in  CTL  formulas  are 
then  restricted  to  fair  paths.  In  the  remainder  of  this  section  we  describe  how  to  modify 
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the  algorithm  above  to  handle  fairness  constraints.  We  assume  the  fairness  constraints  are 
given  by  a  set  of  CTL  formulas  H  =  {hi, . . . ,  hn}.  We  define  a  new  procedure  CheckFair 
for  checking  CTL  formulas  relative  to  the  fairness  constraints  in  H .  We  do  this  by  giving 
definitions  for  new  intermediate  procedures  CheckFairEX ,  CheckFairEU ,  and  Check  Fair  Ed 
which  correspond  to  the  intermediate  procedures  used  to  define  Check. 

Consider  the  formula  EG/  given  fairness  constraints  El.  The  formula  means  that  there 
exists  a  path  beginning  with  the  current  state  on  which  /  holds  globally  (invariantly)  and 
each  formula  in  H  holds  infinitely  often  on  the  path.  The  set  of  such  states  S'  is  the  largest 
set  with  the  following  two  properties: 

1.  all  of  the  states  in  S  satisfy  /,  and 

2.  for  all  fairness  constraints  hk  €  H  and  all  states  s  €  S ,  there  is  a  sequence  of  states  of 
length  one  or  greater  from  s  to  a  state  in  S  satisfying  hk  such  that  all  states  on  the 
path  satisfy  /. 

It  is  easy  to  show  that  if  these  conditions  hold,  each  state  in  the  set  is  the  beginning  of  an 
infinite  computation  path  on  which  /  is  always  true,  and  for  which  every  formula  in  //  holds 
infinitely  often.  Thus,  the  procedure  CheckEairEG{f{v))  will  compute  the  greatest  fixpoint 

gfp  Z(v)  [f(v)  A  f\  CheckEX ( CheckEU  { f  [v) ,  Z(v)  A  Check{hk)))\. 

k= i 

The  fixed  point  can  be  evaluated  in  the  same  manner  as  before.  The  main  difference  is  that 
each  time  the  above  expression  is  evaluated,  several  nested  fixed  point  computations  are 
done  (inside  CheckEU). 

Checking  EX  /  and  E[/  U  g]  under  fairness  constraints  is  simpler.  The  set  of  all  states 
which  are  the  start  of  some  fair  computation  is 

fa  ir{  v )  =  Cli  eck  Fair)  EGT  ru  e ) . 

The  formula  EX  /  is  true  under  fairness  constraints  in  a  state  .s  if  and  only  if  there  is  a 
successor  state  .s'  such  that  s'  satisfies  /  and  .s'  is  at  the  beginning  of  some  fair  computation 
path.  It  follows  that  the  formula  EX  /  (under  fairness  constraints)  is  equivalent  to  the 
formula  EX(/  A  fair)  (without  fairness  constraints).  Therefore,  we  define 

CheckFairEX (f  (v))  =  CheckEX (f(v)  A  fair(v)). 

Similarly,  the  formula  E[/  U  g]  (under  fairness  constraints)  is  equivalent  to  the  formula 
E[/U  ( g  A  fair)]  (without  fairness  constraints).  Hence,  we  define 

CheckFairEU  [f{v),g{v))  —  CheckEU  (f{v),  g(v)  A  fair(v)). 


6.  Counterexamples  and  Witnesses 

One  of  the  most  important  features  of  CTL  model  checking  algorithms  is  the  ability  to 
find  counterexamples  and  witnesses.  When  this  feature  is  enabled  and  the  model  checker 
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determines  that  a  formula  with  a  universal  path  quantifier  is  false,  it  will  find  a  computation 
path  which  demonstrates  that  the  negation  of  the  formula  is  true.  Likewise,  when  the  model 
checker  determines  that  a  formula  with  an  existential  path  quantifier  is  true,  it  will  find  a 
computation  path  that  demonstrates  why  the  formula  is  true.  For  example,  if  the  model 
checker  discovers  that  the  formula  AG  f  is  false,  it  will  produce  a  path  to  a  state  in  which 
-,/  holds.  Similarly,  if  it  discovers  that  the  formula  EF  /  is  true,  it  will  produce  a  path  to 
a  state  in  which  /  holds.  Note  that  the  counterexample  for  a  universally  quantified  formula 
is  the  witness  for  the  dual  existentially  quantified  formula.  By  exploiting  this  observation 
we  can  restrict  our  discussion  of  this  feature  to  finding  witnesses  for  the  three  basic  CTL 
operators  EX,  EG,  and  EU. 

We  start  by  considering  the  complexity  of  finding  a  good  witness  for  the  formula  EG  / 
under  the  set  of  fairness  constraints  H  =  {hx,...,hn}.  We  will  identify  each  hi  with  the 
set  of  states  that  make  it  true.  Given  a  state  5  satisfying  EG/,  we  must  exhibit  a  path 
7r  starting  with  s,  such  that  /  holds  at  each  state,  and  every  fairness  constraint  h  G  H  is 
satisfied  infinitely  often  along  the  path  ir.  Since  the  witness  is  an  infinite  path,  we  must  find 
a  finite  representation  for  it.  It  is  easy  to  see  that  a  witness  can  always  be  found  that  consists 
of  a  finite  prefix  followed  by  a  repeating  cycle.  Each  fairness  constraint  hi  is  satisfied  at  least 
once  on  the  cycle.  Such  a  path  is  called  a  finite  witness.  The  length  of  a  finite  witness  is 
defined  as  the  total  length  of  the  prefix  and  the  cycle.  It  is  desirable  to  find  a  finite  witness 
with  minimal  length;  however,  this  problem  is  NP-complete. 


Theorem  1  If  fairness  constraints  are  permitted ,  finding  a  finite  witness  with  minimal 
length  for  the  formula  EG  True  is  NP-complete. 

Proof:  It  is  relatively  easy  to  see  that  this  problem  in  NP.  The  prefix  of  a  minimal  finite 
witness  cannot  contain  a  cycle,  so  its  length  is  bounded  by  the  number  of  states.  The  cycle 
of  a  minimal  finite  witness  can  be  decomposed  into  several  simple  cycles.  Each  simple  cycle 
must  contain  a  state  that  satisfies  a  fairness  constraint  that  does  not  hold  in  any  other  simple 
cycle.  Otherwise,  we  can  eliminate  this  simple  cycle  from  the  witness.  The  length  of  the 
complete  cycle  is  therefore  bounded  by  the  product  of  the  number  ol  fairness  constraints 
and  the  number  of  states.  Consequently,  it  is  possible  to  guess  a  prefix  and  cycle  and  check 
to  see  whether  they  constitute  a  minimal  finite  witness  in  polynomial  time  in  the  size  ol  the 
graph. 

Finding  a  Hamiltonian  cycle  for  a  directed  graph  is  known  to  be  an  NP-complete  problem. 
Thus,  it  is  sufficient  to  prove  that  the  Hamiltonian  cycle  problem  can  be  reduced  to  the 
minimal  finite  witness  problem.  Consider  an  instance  of  the  Hamiltonian  cycle  problem  for 
a  directed  graph  with  n  nodes.  This  graph  is  treated  as  a  state-transition  graph  and  the 
set  of  fairness  constraints  H  =  {hi,  •  •  • ,  hn }  is  selected  so  that  each  state  satisfies  a  distinct 
fairness  constraint.  On  any  finite  witness,  each  state  must  appear  at  least  once  on  the  cycle; 
hence,  the  length  of  the  finite  witness  must  be  at  least  n.  The  length  ol  the  minimal  finite 
witness  is  n  if  and  only  if  the  n  states  on  the  path  form  a  Hamiltonian  cycle.  Thus,  the 
Hamiltonian  cycle  problem  reduces  to  finding  a  minimal  finite  witness  and  checking  if  this 
path  has  length  n.  This  reduction  can  be  performed  in  polynomial  time.  Consequently,  the 
minimal  finite  witness  problem  is  also  NP-complete.  □ 
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Although  we  are  unable  to  find  the  minimal  finite  witness  easily,  we  still  want  to  obtain 
a  finite  witness  that  is  as  short  as  possible.  In  order  to  accomplish  this  task,  we  will  need  to 
examine  the  strongly  connected  components  of  the  transition  graph  determined  by  the  Kripke 
structure.  We  will  say  that  two  states  Sj  and  s2  are  equivalent  if  there  is  a  path  from  Si  to  s2 
and  also  from  s2  to  si.  We  will  call  the  equivalence  classes  of  this  relation  strongly  connected 
components.  We  can  form  a  new  graph  in  which  the  nodes  are  the  strongly  connected 
components  and  there  is  an  edge  from  one  strongly  connected  component  to  another  if  and 
only  if  there  is  an  edge  from  a  state  in  one  to  a  state  in  the  other.  It  is  easy  to  see  that  the 
new  graph  does  not  contain  any  proper  cycles,  i.e.,  each  cycle  in  the  graph  is  contained  in 
one  of  the  strongly  connected  components.  Moreover,  since  we  only  consider  finite  Kripke 
structures,  each  infinite  path  must  have  a  suffix  that  is  entirely  contained  within  a  strongly 
connected  component  of  the  transition  graph. 

Recall  that  the  set  of  states  that  satisfy  the  formula  EG  /  with  the  fairness  constraints 
H  is  given  by  the  formula 

gfpZ[/A  /\EX(E[/UZA/i,])]  (1) 

k=i 

For  brevity,  we  will  use  EG  /  to  denote  the  set  of  states  that  satisfy  EG  /  under  the  fairness 
constraints  H .  We  construct  the  witness  path  incrementally  by  giving  a  sequence  of  prefixes 
of  the  path  of  increasing  length  until  a  cycle  is  found.  At  each  step  in  the  construction  we 
must  ensure  that  the  current  prefix  can  be  extended  to  a  fair  path  along  which  each  state 
satisfies  /.  This  invariant  is  guaranteed  by  making  sure  that  each  time  we  add  a  state  to 
the  current  prefix,  the  state  satisfies  EG/. 

First,  we  evaluate  the  above  fixpoint  formula.  In  every  iteration  of  the  outer  fixpoint 
computation,  we  compute  a  collection  of  least  fixpoints  associated  with  the  formulas  E[/U 
Z  A  h],  for  each  fairness  constraint  h  6  //.  For  every  constraint  h ,  we  obtain  an  increasing 
sequence  of  approximations  Qq,  Q\,Q2, . . .,  where  Q j*  is  the  set  of  states  from  which  a  state 
in  Z  A  h  can  be  reached  in  i  or  fewer  steps,  while  satisfying  /.  In  the  last  iteration  of  the 
outer  fixpoint  when  Z  =  EG /,  we  save  the  sequence  of  approximations  Qh  for  each  h  in  11 . 

Now,  suppose  we  are  given  an  initial  state  .s  satisfying  EG  /.  Then  .s  belongs  to  the  set 
of  states  computed  in  equation  (1),  so  it  must  have  a  successor  in  E[/  U  (EG/)  A  //]  for 
each  h  €  H .  In  order  to  minimize  the  length  of  the  witness  path,  we  choose  the  first  fairness 
constraint  that  can  be  reached  from  s.  This  is  accomplished  by  testing  the  saved  sets  Q ■  for 
increasing  values  of  i  until  one  is  found  that  contains  some  successor  f  of  s.  Note  that  since 
t  €  Qf,  it  has  a  path  to  a  state  in  (EG/)  A  h  and  therefore  /  is  in  EG/.  If  i  >  0,  we  find  a 
successor  of  t  in  Qi_x-  This  is  done  by  finding  the  set  of  successors  of  t.  intersecting  it  wit  h 
Qi_i ,  and  then  choosing  an  arbitrary  element  of  the  resulting  set.  Continuing  until  i  =  0,  we 
obtain  a  path  from  the  initial  state  s  to  some  state  in  (EG /)  A  h.  We  then  eliminate  h  from 
further  consideration,  and  repeat  the  above  procedure  until  all  of  the  fairness  constraints 
have  been  visited.  Let  s'  be  the  final  state  of  the  path  obtained  thus  far. 

To  complete  a  cycle,  we  need  a  non-trivial  path  from  s'  to  the  state  t  along  which  each 
state  satisfies  /.  In  other  words,  we  need  a  witness  for  the  formula  {.s'}  A  EXE[/U  {/}].  If 
this  formula  is  true,  we  have  found  the  witness  path  for  s.  This  case  is  illustrated  in  Figure 
1.  If  the  formula  is  false,  there  are  several  possible  strategies.  The  simplest  is  to  restart  the 
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Figure  1:  Witness  is  in  first  strongly  connected  component 

procedure  from  the  final  state  s'.  Since  {s'}  A  EXE[/U  {<}]  is  false,  we  know  that  s'  is  not 
in  the  strongly  connected  component  of  /  containing  t,  however  s'  is  in  EG/.  Thus,  if  we 
continue  this  strategy,  we  must  descend  in  the  directed  acyclic  graph  of  strongly  connected 
components,  eventually  either  finding  a  cycle  tv,  or  reaching  a  terminal  strongly  connected 
component  of  /.  In  the  latter  case,  we  are  guaranteed  to  find  a  cycle,  since  we  cannot  exit 
a  terminal  strongly  connected  component.  This  case  is  illustrated  in  Figure  2. 

A  slightly  more  sophisticated  approach  would  be  to  precompute  E[(EG./)  U  {t}].  The 
first  time  we  exit  this  set,  we  know  the  cycle  cannot  be  completed,  so  we  restart  from  that 
state.  Heuristically,  these  approaches  tend  to  find  short  counterexamples  (probably  because 
the  number  of  strongly  connected  components  tends  to  be  small),  so  no  attempt  is  made  to 
find  the  shortest  cycle. 

The  witness  procedure  for  EG  /  under  fairness  constraints  H  can  be  used  to  extend 
witnesses  for  E[/  U  g]  and  EX  /  to  infinite  fair  paths.  Let  fair  be  the  set  of  states  that 
satisfy  EG  True  under  the  fairness  constraints  H.  We  can  compute  E[./  U  g]  under  H  by 
using  the  standard  CTL  model  checking  algorithm  (without  fairness  constraints)  to  compute 
E [/  U  (g  A  fair)}.  Similarly,  We  can  compute  EX  /  by  using  the  standard  CTL  model 
checking  algorithm  to  compute  EX(/  A  fair). 

In  order  to  test  the  procedure  for  finding  counterexamples  when  fairness  constraints  are 
used,  we  have  examined  an  error  in  an  arbiter  design  originally  developed  by  Seitz  [12].  The 
circuit  is  shown  in  Figure  3;  it  is  designed  to  be  speed  independent,  which  means  that  each 
gate  can  take  an  arbitrarily  long  time  to  respond  to  its  inputs.  Fairness  constraints  are  used 
to  ensure  that  every  gate  eventually  responds. 

An  attempt  was  made  to  verify  the  circuit  using  an  explicit  state  model  checker  [7]. 
However,  the  attempt  failed  because  the  number  of  states  was  too  large.  In  order  to  complete 
the  verification,  one  of  the  input  devices  had  to  be  disabled.  By  using  symbolic  model 
checking  techniques,  we  are  able  to  verify  the  original  circuit  without  using  any  simplifying 
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Figure  2:  Witness  spans  three  strongly  connected  components 


TR2  TA2 

Figure  3:  An  asynchronous  arbiter 
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assumptions.  The  model  contains  33,633  reachable  states,  and  the  entire  verification  takes 
only  a  few  minutes. 

We  have  verified  several  liveness  properties  which  require  that  each  request  signal  in¬ 
evitably  leads  to  an  acknowledgement  signal.  Such  properties  can  be  easily  represented  by 
CTL  formulas  with  the  form  AG(r  — *  AF  a),  where  r  represents  a  request  and  a  represents 
an  acknowledgment.  An  error  was  discovered  when  the  specification  AG(frl  — >  AFtal) 
was  checked.  The  algorithm  given  earlier  in  this  section  found  a  counterexample  that  was 
seventy  eight  states  long  and  had  a  cycle  with  length  thirty.  The  counterexample  showed 
that  the  following  execution  sequence  was  possible.  The  circuit  could  reach  a  state  where 
every  node  was  low  except  meol  if  the  ME  element  took  a  long  time  to  respond.  When 
url  was  issued,  trl,  tal,  sr,  sa  and  ual  became  true  consecutively.  Because  of  the  long 
delay  of  the  0R1  gate,  meil  remained  low.  Eventually,  the  ME  element  responded  to  its 
inputs  and  set  meol  low.  This  caused  trl  and  tal  to  become  low.  Next,  OR1  responded 
and  meil  became  high.  Then,  the  ME  element  and  the  AND1  gate  caused  trl  to  become 
high  again  while  tal  continued  to  be  low.  In  this  state,  the  formula  trl  — *■  AF  tal  was  false. 
Since  ual  was  already  high,  url  could  become  low.  This  caused  trl  to  become  low.  The 
counterexample  showed  that  url  was  always  low.  Therefore,  tal  remained  low  as  well.  A 
correction  for  the  error  was  proposed  in  [7],  but  will  not  be  discussed  here. 

7.  Counterexamples  and  Witnesses  for  CTL*  Formulas 

In  the  previous  sections,  we  described  how  to  perform  model  checking  and  find  counterexam¬ 
ples  or  witnesses  for  CTL  formulas.  However,  some  temporal  properties  that  are  important 
for  reasoning  about  sequential  circuit  designs  and  protocols  cannot  be  expressed  by  CTL  for¬ 
mulas.  In  these  cases,  an  extension  of  CTL,  called  CTL*,  is  often  used.  There  are  two  types 
of  formulas  in  CTL*:  state  formulas  (which  are  true  in  a  specific  state)  and  path  formulas 
(which  are  true  along  a  specific  path).  As  before,  let  AP  be  the  set  of  atomic  propositions. 
The  syntax  of  state  formulas  is  given  by  the  following  rules: 

•  If  p  €  AP,  then  p  is  a  state  formula. 

•  If  f  and  g  are  state  formulas,  then  ->/  and  f  V  g  are  state  formulas. 

•  If  /  is  a  path  formula,  then  E(/)  is  a  state  formula. 

Two  additional  rules  are  needed  to  specify  the  syntax  ot  path  formulas: 

•  If  /  is  a  state  formula,  then  /  is  also  a  path  formula. 

•  If  /  and  g  are  path  formulas,  then  ->/,  /  V  g,  X  /,  and  /  U  g  are  path  formulas. 

CTL*  is  the  set  of  state  formulas  generated  by  the  above  rules.  The  logical  connectives 
and  V  have  their  usual  meaning.  The  formula  E (/)  is  true  in  a  state  when  there  exists  a 
path  from  the  state  such  that  /  holds  along  the  path.  Let  tt  =  s0,  si, . . .  be  a  path.  We  use 
7 t1  to  denote  the  suffix  of  7r  starting  at  .s,.  A  state  formula  holds  along  tt  when  it  is  true  in 
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the  first  state  s0.  X  /  holds  along  r  when  /  holds  along  7T1.  Finally,  the  formula  f\Jg  holds 
along  7 r  when  there  exists  a  k  >  0  such  that  g  holds  on  and  /  holds  along  every  7rJ  where 
0  <  j  <  k.  The  following  abbreviations  are  used  in  writing  CTL*  formulas: 

•  /  A  </  =  V  ~'g)  •  F  /  =  true  U  / 

•  A  (/)  =  -E(-/)  •  G  /  =  -■  F  ->/ 

In  general,  model  checking  is  very  expensive  for  CTL*  formulas.  However,  for  a  large 
class  of  formulas  which  have  the  form  E  V"-i  AjLi(GF pS]  V  FG  qtJ),  efficient  model  checking 
algorithms  exist  [8].  Because 

E  V  A  (GF  P«  V  FG  q,, )  =  V  E  A  (GF  Pil  V  FG 

i=ij=i  i=i  j=i 

it  is  sufficient  to  check  formulas  having  the  form  EAj=i(GF Pj  V  FG(/j).  A  fixed  point 
characterization  for  these  formulas  is  given  in  [8] 

E  A  (GF  Pj  V  FG  r/A  =  EF  gfp  Y  [/\((</j  A  EX  V)  V  EX  E[V  U(WA  V)]  )]• 
i=i  j= o 

By  performing  a  computation  that  is  similar  to  the  one  described  in  Section  5,  we  are  able 
to  check  the  restricted  class  of  CTL*  formulas  mentioned  above.  The  problem  of  finding 
witnesses  for  these  formulas  is  more  complicated.  Suppose  that  we  want  find  a  witness  for 
so  f=  E  A"-i(GF pj  V  FGc/j).  It  is  easy  to  see  that 

E/\(GFPjvFG,,) 

j=i 

n—  1 

=  E  A  (GF p3  V  FG qj)  A  (GF pn  V  FG qn) 

j= i 

n  —  1  \  /  n  —  1 

E  A  (GF pj  V  FG  q j )  A  GF  pn\  V  (  E  A  (GF pj  V  FG  </,)  A  FG  r/„ 

Consec[uently,  if  so  f=  EAj=/(GF  Pj  V  FG(/J  A  FG  </„,  it  is  sufficient  to  find  a  witness  for 

this  formula;  otherwise,  a  witness  must  exist  for  E  Aj=i' (GF  p:  V  FG  q] )  A  GF  p„.  If  \\v 
continue  this  process  for  the  remainder  of  the  formula,  we  will  eventually  obtain  a  formula 
which  has  the  form  E  FG  r/,-,  A  ...  A  FG  qlk  A  GF  pJX  A  ...  A  GF  Pjn-k  ■  Because 

E(FG  r/i,  A. .  .AFG  qlk  AGF  pJ1  A. . .  AGF  pJn_k )  =  EF  EG  (</;,  A. .  .A^  AF  A...AF  /A„_t ). 

this  formula  is  true  if  and  only  if  the  CTL  formula  EG(f/M  A  ...  A  <gk )  is  true  under  the 
fairness  constraints  pji , . . . ,  Pj„_k .  A  witness  can  be  computed  in  exactly  the  same  manner 
as  in  the  last  section. 
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8.  Counterexamples  for  Language  Containment  Problems 

An  alternative  technique  for  verifying  finite-state  systems  is  based  on  showing  language  in¬ 
clusion  between  finite  o;-automata  [9,  10,  13].  We  model  the  system  to  be  verified  by  an 
w-automaton  Ksys.  The  specification  to  be  checked  is  given  by  a  second  cu-automaton  I<spec. 
The  system  will  satisfy  its  specification  if  the  language  accepted  by  Ksys  is  contained  in 
the  language  accepted  by  Kspec,  i.e.  C(I<sys)  C  C(Kspec).  In  this  section  we  show  how  the 
techniques  described  in  last  section  can  be  used  to  find  counterexamples  for  language  con¬ 
tainment  problems.  Although  there  are  many  types  of  u;-automata,  in  this  paper  we  only 
consider  Streett  automata.  These  automata  are  particularly  useful  for  modeling  systems 
with  complicated  fairness  constraints  that  cannot  be  handled  using  the  technique  described 
in  Section  5.  Counterexamples  for  other  types  of  a>-automata  can  be  determined  in  a  similar 
manner  by  using  results  from  [4].  In  general,  checking  language  inclusion  between  two  non- 
deterministic  w-automata  is  PSPACE-hard.  For  this  reason  we  require  that  the  specification 
automaton  be  deterministic.  We  require  that  both  automata  be  complete. 

A  ( nondeterministic )  ^-automaton  is  a  5-tuple  I<  =  (S,  s0,  E,  A,  F),  where 

•  S  is  a  finite  set  of  states 

•  So  £  S  is  the  initial  state 

•  E  is  a  finite  alphabet 

•  ACS'xExS'is  the  transition  relation 

•  F  is  the  acceptance  condition. 

The  automaton  is  deterministic  if  for  all  states  s,tut2  G  S  and  input  symbols  a  £  S, 
if  (s,<r,ti)  and  (s,<r,t2)  are  two  transitions  in  A,  then  tx  =  t2.  The  automaton  is  complete 
if  for  every  state  s  £  S  and  for  every  symbol  cr  £  S,  there  is  a  state  .s'  £  S  such  that 
(s,cr,  s')  £  A.  An  infinite  sequence  of  states  SoSjS2  . . .  £  Sw  is  a  run  of  an  ^-automaton  if 
there  exists  an  infinite  sequence  cr0cr\cr2 . . .  €  S1"  such  that  Vi  >  0,  (s,,  cr.,-,  s,+i )  £  A.  The 
infinitary  set  of  a  sequence  S0S1S2  ...  £  Su,  denoted  by  inf(soSi  . . .),  is  the  set  of  all  the  states 
that  appear  infinitely  many  times  in  the  sequence.  The  Streett  acceptance  condition  has  the 
form  F  =  {(C/i,  Vi), . .  • ,  {Un,  Vn)},  where  V{  C  S.  A  sequence  <70<ti<t2  ...  £  E1^  is  accepted 
by  a  Streett  automaton  if  there  is  a  corresponding  run  s0SiS2--  -  £  with  the  property 
that  for  every  i  £  {1, . . .  ,n},  inf(r)  C  Ut  or  inf(r)  n  V{  ±  0.  The  set  of  sequences  accepted 
by  an  automaton  M  is  called  the  language  of  M  and  is  denoted  by  C(M). 

Let  I<  =  (5,s0,E,  A,F),  K'  =  (S",4,E,  A',F'}  be  a  pair  of  Streett  automata  over  the 
same  alphabet.  It  is  shown  in  [4]  that  the  path  formula  (j>F  expresses  the  acceptance  condition 

d>F=  A  (FG(V  s)VGF(  \/  s)), 

( U,V)eF  SEU  SEV 

and  that  expresses  the  negation  of  the  acceptance  condition  for  K'\ 

—  V  (GF(  \f  s)  A  FG(  V_5)) 

(U',V')EF<  seW  sEV' 
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Let  M(K,K')  =  (S  x  S',  (sq,s'0),  C,7Z)  be  a  state-transition  system  such  that  jC(s,s')  = 
{s,s'}  and  (s,s')TZ(t,t')  (3a  £  E  :  (s,a,f)  £  A  and  (s',a,t')  £  A).  If  K  is  a  nondeter- 

ministic  Streett  automaton  and  K'  is  a  deterministic  Streett  automaton,  then 

C(K)  C  jC(K')  &  M(I\,  I\')  \=  A-uAfO 

where  fp  and  are  the  formulas  given  above.  Note  that  the  above  equivalence  does  not 

hold  if  I\'  is  a  nondeterministic  automaton.  The  formula  E(^f  A  is  equivalent  to 

(  A  (FG(  \/  s)  VGF(  \/  •*))]  A  (  V  (GF(V«)AFG(  V  «))) 

\(u,v)eF  sen  sev  )  \(U',v')eF'  seU7  sev^  ) 

=  V  ((  A  (fg(Vs)vgf(Vs)))  AGF(  V  '^)AFG(  V  A- 

(U',v')eF'  \  \(U,v)eF  seu  *ev  /  s^zf7  sei7r  J 

This  formula  is  an  instance  of  the  CTL*  formulas  discussed  in  last  section.  Thus,  the 
technique  given  in  last  section  for  finding  witnesses  can  be  used  to  find  a  counterexample 
when  C(K)  is  not  contained  in  C(K').  Counterexamples  for  the  language  inclusion  problems 
of  Biichi,  Muller,  Rabin,  and  L  automata  can  be  found  in  essentially  the  same  way. 

9.  Directions  for  Future  Research 

In  this  paper,  we  have  described  an  efficient  technique  for  generating  counterexamples  and 
witnesses  for  symbolic  model  checking  algorithms.  However,  when  the  number  of  reachable 
states  is  very  large,  the  counterexample  can  still  be  very  long.  Techniques  for  generating 
even  shorter  counterexamples  will  make  symbolic  model  checking  more  useful  in  practice. 

Finding  a  counterexample  can  sometimes  take  most  of  the  execution  time  required  for 
model  checking.  Additional  research  is  needed  to  develop  more  efficient  algorithms.  This  is 
particularly  important  because  the  model  checking  algorithm  may  need  to  be  invoked  several 
times  in  order  to  find  the  witness  for  a  CTL*  formula. 

Another  problem  with  the  counterexample  generated  by  the  model  checker  is  that  it  is 
sometimes  hard  to  read.  A  more  readable  form  will  be  helpful  to  engineers  who  are  not 
familiar  with  model  checking. 
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